“Data is Information is Knowledge is Wisdom.” This truism was epitomized by Russel Ackoff in the year 1989. However, the data processing community has been tackling the intricacies associated with this pipeline for decades. Intellectual exercises have resulted in more than one ways of transitioning from one level (say data) to the other (say information). The levels are typically bridged using combinations of the likes of logical, mathematical, statistical, and other xxxxa1 tools typically referred to as algorithms, heuristics, cookbooks etc. (collectively algorithms hereafter). The level bridgers are referred to as tasks and the complete pipeline—an application. So an application is a collection of tasks each of which can be accomplished with multiple competing algorithms that have unique characteristic response to the ever-varying input data.
Depending upon the assumptions made about the levels and the intrinsic, extrinsic capability of the algorithm bridging the levels, the conclusion arrived at could be far from ideal. Hence it is essential to ascertain that the final conclusion arrived at is consistent with the given input data. This process is known as validation. This is done using a number of metrics, each of which typically involves comparing the result with a ground truth or surrogate truth. There are two main issues with this approach. Firstly, the metrics computed in the validation process is heavily dependent on the task to be accomplished. This makes the validation more algocentric instead of the preferred datacentric. Secondly, the ground truth holds good only for the given instance of the input data. As such if the ground truth is critical to the validation process, and it can be arrived at, irrespective of the difficulty in getting it, what is the need for exploring with a plethora of algorithms? Hence the present approach to validate applications seems to be in a quagmire.
Any new field typically undergoes a hype cycle and levels down finally to something that is useful and practical. That process ultimately converges on a recipe for mixing and matching the metrics to arrive at reasonable validation approaches. It is unlikely that such a final approach would be anything similar to the current practice.
FIG. 1 shows the current processing-validation (PV) pipeline. In this approach, the validation is done at the final stage after the data processing is completed. This is a necessary but insufficient step. Doing so ignores the fact that the application is a chain of tasks and that a chain is only as strong as the weakest link. This approach does not attempt to identify the weakest link. FIG. 2 shows a logical next-step approach.
In the refined PV pipeline, the individual tasks are validated. This helps to identify and address the weakest chain in the task link. This pipeline could be further refined by accommodating a number of independent ways (algorithms) to achieve the task. Even in the (debatable) absence of competing algorithms, there could be controllable parameter within the same algorithm that could respond differently to the input data.
By way of example, assume that an application has two tasks and for each task there are four competing algorithms. The corresponding PV pipeline is given in FIG. 3. If Ai is the number of algorithms supported at each task, the number of input paths to the validation step is
      ∏          i      =      1              #      ⁢      tasks        ⁢      A    i  Just for a two stage, 4 algos/stage application, this would be 16. Hence it is practically infeasible to implement such a pipeline. Using a scheme similar to that in FIG. 2, the number of validations to be performed would be the sum of the number of algorithms supported in each stage. Such a pipeline is shown in FIG. 4.
The interleaved PV pipeline shown in FIG. 4 makes logical sense since no serious application builder will restrict an application to be driven by a single algorithm for each critical task. Moreover, each algorithm responds uniquely to the input data. So an approach to validate the results at the individual task level across multiple competing algorithms allows the data to be processed by the best algorithm suited for it.
Though this might seem logically sound, it is not practically feasible due to the following bottlenecks:                1. For the PV pipeline to support assembly line deployment, the validation task should be fully automatic. Unfortunately, there hardly seems to be any single scheme that provides this for any task.        2. Applications are often built by cascading off-the-shelf APIs (application programmer's interface). The validation approach and the metrics to be calculated therein vary depending on the task being performed. So the application builder is burdened with the job of writing validation functions for each task.        3. Even if the application programmer stubs the validation functions, as mentioned before, majority of the prevailing metrics involve using ground truth which is almost impossible to obtain at the individual task level.        4. Validation has to be performed “live”. Unfortunately, existing validation methodologies take more time, interaction and tweaking at run time than, at times, the original task level algorithms. Even if multiple algorithm execution is feasible for each task, overheads caused by validation could be prohibitively expensive.        5. Existing validation strategies are time consuming, since they typically involve computing sufficient number of metrics and suitably weighting them. Finding the right weights is more of an art than science. Moreover, at the task validation level shown in FIG. 5, a simple “yes” or “no” answer is needed to the question “Given the input data, is algorithm x.y better than x.Y for this application?” Unfortunately, current metrics do not support such a simple answer.        
Hence there is a need for a task validation approach that is robust, fully automatic, fast, scaleable, simple, data and task independent and is not dependent on the availability of ground or surrogate truth.